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(54) Method and system for creating dental models from imagery 



(57) Creating a dental model from a series of images 
of an intra-oral object includes the steps of (a) capturing 
a series of images of an intra-oral object from a plurality 
of capture positions, where the object includes common 
surface features and a control target arranged with re- 
spect to the object to provide control features; (b) meas- 
uring the common features from the series of images of 
the object and the control features from the control tar- 
get imaged with the images of the object; (c) analytically 



generating a 3-dimensional model of the object by pho- 
togrammetrically aligning the measurements of the con- 
trol features, thereby reducing image errors due to the 
variability of the capture positions; and (d) adjusting the 
photogrammetrically aligned 3-dimensional model of 
the object by aligning the common features of the model 
to like features on the image of the object, thereby pro- 
ducing an aligned dental model from the series of imag- 
es. 



CM 

O 
(D 

CM 



Q. 

UJ 



MULTIPLE IMAGES OF 
OBJECT WITH CONTROL TARGET 



12- 



20- 



REFiNE 
PHOrOGRAMMETRtC 
ADJUSTMENT 



28 — 



24 



\ NC 
"ACCEPT^ 



PROJECT 
5-D 
MODEL 



DETERMINE 
IHS-ALJCNA4ENT 
OF CONTROL 



rES 




3d 




DETERMINE 




i4iS~ ALIGNMENT 






OF 3- 


D hiOOEL 





42-^ 




40- 



NO 



MEASURE 
COMMON 
FEATURES 



CREATE 

3-D 
MODEL 



ADJUST 

3-D 
MODEL 



APPLY 
3-D 
MORPHING 



22- 



MEA5URE 
CONTROL 
FEATURES 



PHOTO- 
GRAMME TRIC 
ADJUSmENT 



FIG. 1 



PROJECT 
3-D 
MODEL 



45^ 



DETERMINE 
MIS-AliCNMENT 
OF 3-0 MODEL 




Printed by Jouve. 75001 PARIS (FR) 



BNSDOCID: <EP 1276072A1 J_> 



1 



EP 1 276 072 A1 



2 



Description 

FiELD OF THE INVENTION 

[0001] The invention relates generally to the field of 

dental imagery, and in particular to a method and appa- 
ratus for effecting imagery of a prepared cavity in a tooth 
followed by automatic generation of a model to control 
automatic fabrication of a dental inlay for the cavity. 

BACKGROUND OF THE INVENTION 

[0002] The invention described herein relates gener- 
ally to the following conventional situation. A dentist pre- 
pares a cavity of a decayed tooth to allow its restoration 
by means, e.g., of an inlay or a crown. After the prepa- 
ration has been rendered, an impression of the cavity is 
taken, and is ordinarily sent to a dental laboratory. Con- 
trary to such conventional techniques, there are differ- 
ent and more recent techniques which alleviate the role 
of the dental laboratory and fabricate the desired restor- 
ative piece in the dental office. In particular, the prepared 
cavity is registered by an electro-optic scan head. The 
data thus obtained can be complemented by operator 
input, using techniques from the CAD {Computer-Aided- 
Design) domain, and the final piece is fabricated with 
the aid of a miniature NC (numerical control) grinding 
machine. 

[0003] U.S. Patent No. 4,837,732 (Brandestinl et al) 
describes a method for a dentist to record the shape in 
situ of teeth prepared for repair. The method involves 
the acquisition of data defining the three-dimensional 
shape of prepared teeth and their immediate vicinity. 
First, a video display shows a live image from a scan 
head, and the scan head is manually oriented relative 
to the prepared teeth while observing the image of the 
teeth on the video display. Thereafter the data produced 
by the scan head in a selected orientation generates 
corresponding depth and contrast images, and a depth 
image is processed based on the contrast image. This 
method also includes the step of superimposing graphic 
markers on the image displayed on the video display to 
facilitate an on-line alignment of the teeth displayed in 
the live Image with reference data from previous data 
acquisitions. 

[0004] The drawback to this method from the prior art 
is that it incorporates a registration scheme that can later 
interfere with the quality of the results, and also requires 
that the dentist be able to hold the scan head almost 
perfectly still at a specific point in the procedure. More 
specifically, the artifacts typlcafly due to the 3D registra- 
tion scheme (such as fringe, speckle and/or Venetian 
blind effect) are cited in the patent as "intolerable and 
must be eliminated" since phase angle differences are 
used for measurement of the depth. Furthemore, the 
patent cites a need for a "quasi-instantaneous 3D ac- 
quisition following a trigger release", the essential con- 
dition being that the orientation of the scan head must 



not change between the search and acquisition modes. 
[0005] What happens in Brandestinl et al is that the 
3D result is overlaid on the search image allowing the 
dentist to verify the result. What is needed, however, is 
5 a system in which the 3D results are projected mto the 
image using the projective equations of photogramme- 
try. This would cause the results to appear as if they 
were actually present In the scene at the time of image 
acquisition, allowing a much more accurate and precise 
10 evaluation. 

[0006] Previous photogrammetric-based approach- 
es, however, (see, e.g., U.S. Patent No. 5,372,502) 
have not been too successful for a number of reasons. 
For example, such approaches have not been success- 
ful because it is hard to determine the exact relationship 
between the camera and the object. Moreover it is also 
hard to precisely measure the object because teeth are 
fairly unlfomn in color and have little texture (which is in 
part why it is hard to determine the relationship dis- 
cussed above). 

SUMMARY OF THE INVENTION 

[0007] The present invention is directed to overcom- 
ing one or more of the problems set forth above. Briefly 
summarized, according to one aspect of the present in- 
vention, a method (and system) for creating a dental 
model from a series of images of an Intra-oral object In- 
cludes the steps of (a) capturing a series of images of 
the intra-oral object from a plurality of capture positions, 
where the object includes common surface features and 
a control target arranged with respect to the object to 
provide control features; (b) measuring the control fea- 
tures from the control target imaged with the images of 
the object; (c) analytically generating a 3-dimensional 
model of the object by photogrammetrically aligning the 
measurements of the control features, thereby providing 
a photogrammetrically-aligned 3-dimensional model of 
the object while reducing image errors due to the varia- 
ble orientation of the capture positions; and (d) adjusting 
the photogrammetrically aligned 3-dimensional model 
of the object by aligning the common features of the 
model to like features on the image of the object, thereby 
producing an aligned dental model from the series of 
Images. In practfce, the last stage involves the applica- 
tion of a 3-dimensional morphing algorithm to correct for 
the misalignment. 

[0008] The principal advantage of the invention is that 
the use of photogrammetric projection methods and ad- 
justment to control eliminates the need for a registration 
scheme, such as that used in Brandestinl et al, which 
projects stripes of light onto the target and can result in 
unacceptable artifacts. Furthermore, under the present 
invention, there is no need to restrict the acquisition of 
the image(s) to a "quasi-instantaneous" state, as phase 
information is not used. 

[0009] It becomes possible to measure the exact re- 
lationship between the camera and the intra-oral object 
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because the use of a target provides something to 
measure, thus allowing the determination of the relation- 
ship between the camera and target. The use of 3D mor- 
phing addresses the matter of precisely measuring the 
object itself by projecting the data that is available into 
the picture and letting one see how well it fits - In the 3D 
object space. If correct, the model of the tooth should 
"fit" the tooth "skin tight". 

[0010] These and other aspects, objects, features 
and advantages of the present invention will be more 
clearly understood and appreciated from a review of the 
following detailed description of the preferred embodi- 
ments and appended claims, and by reference to the 
accompanying drawings. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0011] 

FIG. 1 is a block diagram of the method for creating 
dental models from imagery according to the inven- 
tion. 

FIG. 2 is a perspective diagram of a target useful 

with the method described in Figure 1 . 

FIG. 3 is a block diagram of a morphing technique 

utilizing a database of generic tooth models. 

FIG. 4 is a diagram of a dental system that utilizes 

the method shown in Figure 1 . 

DETAILED DESCRIPTION OF THE INVENTION 

[0012] Because dental image processing systems 
and methods are well known, the present description will 
be directed in particular to elements forming part of, or 
cooperating more directly with, apparatus and method 
in accordance with the present invention. Elements not 
specifically shown or described herein may be selected 
from those known in the art. Certain aspects of the em- 
bodiment to be described may be provided in software. 
Given the system as shown and described according to 
the invention in the following materials, software not 
specifically shown, described or suggested herein that 
is useful for implementation of the invention is conven- 
tional and within the ordinary skill in such arts. This is 
particularly true given the advanced technrcal state of 
conventional photogrammetry and the well-understood 
current automation of the photogrammetric process. 
[0013] Referring Initially to Figure 4. a preferred em- 
bodiment of the invention is implemented in a system 
including an intra-oral camera 2, a computer system 3 
including instructions for implementing the invention 
and a machine tool 4. In the schematic shown in Figure 
4, it should be understood that the interconnections be- 
tween the camera 2, the computer system 3 and the ma- 
chine tool 4 are shown by arrows, and therefore not spe- 
cifically indicated. These interconnections may take var- 
ious forms, such as a cable or any other electromagnetic 
connection (such as an rf transmission), or the manual 



4 

transfer of data from machine to machine. The camera 
2 may be any type of conventional dental camera that 
is capable of capturing a reasonably high resolution im- 
age of an intra-oral object, such as the teeth 4; a pre- 

5 ferred example is the intra-oral camera disclosed In 
commonly assigned, copending U .S. Patent Application 
Serial No. 09/796,239, entitled "Intra-Oral Camera with 
Integral Display", filed February 28, 2001 in the names 
of J. P. Spoonhower, J.R. Squilla and J.T. Boland, and 

10 which is incorporated herein by reference. 

[0014] The camera 2 is hand held by the dentist and 
several images are captured of the teeth; it is under- 
stood, however, that the orientation of the camera rela- 
tive to the teeth will vary from one image to the next. The 

15 elimination of the effect of these different orientations on 
the subsequent measurements is one feature of the in- 
vention. The digitized data from the camera 2 is trans- 
ferred to the computer system 3 for processing. The 
methodology of the invention is implemented by the 

20 computer system 3 in its processor 5, and the imaging 
results may be interactively displayed on a monitor 6. 
An operator using a keyboard 7 and/or a mouse 8 can 
manipulate a cursor 9 to perform measurements of the 
type that will be subsequently described. The output 

25 from the computer system 3 is a digitized three-dimen- 
sional surface pattern that is transferred to the machine 
tool 4 as a tool path program for the fabrication of a den- 
tal mold or a restorative piece. The program will direct 
a milling cutter 1 0 in the milling of the tooth mold or the 

30 restorative piece 1 1 from a suitable substrate, for exam- 
ple, ceramic or any other suitable machinable material. 
[0015] The dental imaging method according to the 
invention employs a mensuration method that utilizes 
photogrammetric projection, analytical adjustment to 

35 control and three-dimensional morphing to develop ac- 
curate dental models. Mensuration, in this Instance, re- 
fers to a measurement process involving several steps: 
(1 ) the identification of control points on the digitized im- 
age, (2) the stereoscopic transfer of those points to the 

40 overlapping images upon which they appear, and (3) the 
actual measurement of the image coordinetes of the 
control points. 

[0016] Photogrammetry generally is the science of 
measuring graphically by means of light, and more spe- 

45 cif Ically the science of obtaining reliable measurements 
by means of photographs or other form^^ of imagery, 
such as electronic sensing by a sensor (see generally 
Manual of Photogrammetry, Fourth Edition . American 
Society of Photogrammetry, 1980). Photogrammetric 

50 projection refers to an image projection that uses an an- 
alytical representation of the physical model that de- 
scribes the imaging process of the sensor. The temn pro- 
jection specifically refers to the concept of a light ray 
projecting from the intra-oral object, through the sensor 

55 lens, to the image plane, Jn this case using the physical 
model of the imaging process to determine where the 
points will be located. 

[0017] Analytical adjustment to control refers to the 
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process of correcting the set of parameters which de- 
scribe the physical model, to a subset of known, or con- 
trol, parameters. A least squares adjustment process is 
typically applied to a set of normal equations, derived 
from a set of linearized condition equations, which in 
turn are partial derivatives of the image coordinates with 
respect to the total parameter set. Details of the least 
squares process is well known to those of ordinary skill 
in this art and described, e.g., in the Manual of Photo- 
grammetry, Fourth Edition , op. cit., pp. 77 - 88, which is 
incorporated herein by reference. 
[0018] Three-dimensional morphing refers to the 
process of adjusting a 3-dimensional, object model to 
an image(s) of the object. This is accomplished by pro- 
jecting a hypothesized 3-dlmensional model of the ob- 
ject into an existing image (through the analytical phys- 
ical model referred to above), detecting the misalign- 
ment between the true image and the projected, object 
model-derived image, and making corrections to the ob- 
ject model (which Is then re-projected) to improve the 
fit. Techniques for three-dimensional morphing are well 
known in the art and will not be described in detail here- 
in. For further information, reference may be made to 
articles by Frederic Pighin et al., "Synthesizing Realistic 
Facial Expressions from Photographs", in Computer 
Graphics Proceed/rigs, Annual Conference Series, 
1998, pp. 75 - 83 and by TakaakI Akimoto et al., "Auto- 
matic Creation of 3D Facial Models", in IEEE Computer 
Graphics & Applications, September 1 993, pp. 16-22. 
In these articles, the specifics are directed toward facial 
models, but the technology application to teeth models 
would be the same. 

[0019] Referring to Figures 1 and 2, the method ac- 
cording to the invention is shown, in which multiple im- 
ages 12 of an intra-oral object (one or more teeth 14) 
are initially captured from several different aspects and/ 
or positions by the camera 2. For each image, one or 
more of the teeth 1 4 (e.g. , a tooth 1 4a) includes a control 
target 16, as shown In Figure 2. (In practice, the tooth 
is typically either the original, unprepared tooth or the 
tooth as prepared (i.e., a tooth stump) for the restorative 
procedure.) The target 16 Is rigid material, of saddle 
form, whrch rests on the tooth 14a with length C along 
the side of the tooth. (Although not shown as such In 
Figure 2, the control target 1 6 could span several teeth, 
such as both teeth 14a and 14b.) Lengths A, B, C, and 
D are known, and may be unequal. Angles included by 
the vertices 1 8 are also known; as will be described, the 
vertices 18 are the aforementioned known, or control, 
parameters that are used in the analytical adjustment to 
control. Several targets may be constructed in varying 
sizes to accommodate different size teeth. Generally, 
several images are taken from several different aspects/ 
positions as the basis for a 3-dimensional view of the 
intra-oral object, including both the control parameters 
and certain common features on the tooth, such as the 
cusps and valleys describing the natural topographic 
surface of the tooth (or the tooth stump, if the intra-oral 



object is a prepared tooth). 

[0020] The mensuration process involves the meas- 
urement of common features or parameters (the cusps 
and valleys) in a feature measurement stage 20 and the 

5 measurement of control features or paranneters (the ver- 
tices 18) on the target 16 in a control measurement 
stage 22. There are several ways to take these meas- 
urements. Referring to Figure 4, these measurements 
may be interactively taken by an operator positioning the 

to cursor 9 over the respective features on each of the mul- 
tiple images 12 as they are displayed on the monitor 6; 
the coordinates of each measurement are then captured 
by the processor 5. Alternatively, the processor 5 may 
employ appropriate conventional Image processing al- 

t5 gorithms to automatically locate each of the features; 
this may involve image enhancement and other feature 
improvement algorithms, as necessary. The measure- 
ments are then processed In a photogrammetric adjust- 
ment stage 24 in order to compute the object-space co- 

20 ordinates of any object point which Is imaged in the mul- 
tiple overlapping images from varying camera orienta- 
tions; this process utilizes the aforementioned least 
squares process described in the Manual of Photogram- 
metry, Fourth Edition , op. cit. Basically, this is a muftiray 

25 stereo Intersection process that is used to locate each 
image point relative to the camera position. The result 
is a 3-dlmenslonal model 26 of the tooth that has been 
processed with an analytical representation of the phys- 
ical model which represents the imaging process of the 

30 sensor that captured the images. 

[0021 ] However, the imaging device Is usually a hand- 
held camera that does not make a perfect geometric 
representation of the object, which creates enrors due 
to the lack of certainty of knowledge about the image 

35 positions. One of the features of the invention Is to tackle 
the problem of eliminating these errors, i.e., the cam- 
era's variability in orientation, before attempting to cor- 
rect for errors in the actual model of the tooth. In this 
manner, the requirement (and problem) noted In the pri- 

40 or art, namely, that the scan head or imaging device 
must be held perfectly still, can be avoided. Therefore, 
an analytical adjustment using the control points (the 
vertices 1 8) to correct the estimates is made by analyt- 
ically projecting the 3-D model 26 in an analytical pro- 

45 jection stage 28 Into an existing image (one of the mul- 
tiple images 12), detemnining the misalignment of the 
control points (the vertices 1 8) between the model and 
the Image in a misalignment stage 30 and refining the 
photogrammetric adjustments in a refinement stage 32 

50 rf the misalignment Is unacceptable (decision 34). The 
projection Is an analytical process, meaning that it is ac- 
complished mathematically, and the determination of 
misalignment may be accomplished Interactively (by us- 
ing the cursor 9) or automatically with appropriate image 

55 processing algorithms. It is helpful to understand that 
this projection process utilizes the physical model rep- 
resenting the Imaging process, therefore differing from 
a simple overiay of the 3-D model onto the image. Once 
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these corrections are made, the variability in the model 
caused by the various camera orientations is reduced 
to an acceptable level, If not eliminated. 
[0022] Once the control alignment is acceptable, the 
slopes and cuwes between the cusps and valleys in the 
model should either match, or be made to match, the 
corresponding features in the image of the tooth. Thus, 
it is necessary to detemnine the remaining misalignment 
of the model relative to the actual image in a misalign- 
ment determination stage 36, that is, to detemnine the 
misalignment (if any) of the common features in the 
model with respect to the same features in the actual 
image. If misalignment is present (decision 38), a three- 
dimensional morphing stage 40 is initiated for adjusting 
the 3-dimensional object model in an adjustment stage 
42 to an image(s) of the object. (This changes the 3-di- 
mensional position of points in the model without affect- 
ing the prior alignment adjustment regarding the camera 
orientation.) This is accomplished in a projection stage 
44 by projecting the hypothesized 3-dimensional model 
of the object into one of the existing images (through the 
analytical physical model referred to above) of the intra- 
oral object. Then, the misalignment between the true im- 
age and the projected, object model-derived image is 
detected in a misalignment stage 46. If the misalignment 
is acceptable (decision 48), the process is ended; oth- 
erwise, corrections are made to the object model in the 
stages 40 and 42 (which is then re-projected) to improve 
the fit. 

[0023] At this point, an acceptable model of the tooth 
has been generated and may be used in subsequent 
processing, such as in the fabrication of the desired re- 
storative piece, either in a laboratory or in the dental of- 
fice by use of the machine tool 4. It should be understood 
that in addition to teeth, other dental prosthetics can be 
modeled in accordance with the invention, including 
without limitation bridges, veneers and other dental re- 
storative units. Moreover, various other types of fabrica- 
tion may be employed without limitation in addition to 
milling or cutting, such as injection molding. 
[0024] Figure 3 represents an alternative method for 
adjusting the 3-dimensional object model to an image 
(s) of the intra-oral object, that is, an alternative method 
to the logic represented by the elements 40-48 of Figure 
1 . More specifically, in the alternative approach of Figure 
3, generic 3D models from a database 50 of such items 
may be used in addition to, or as a substitute for the 3D 
model provided in the elements 40-48. This approach 
specifically addresses the issue around difficulty in ac- 
curately measuring the cusps and valleys, as they may 
not be as well defined as the target vertices, by allowing 
a generic tooth model to be used instead. This essen- 
tially means that in Figure 1 , the 3-D model 26 can be 
eliminated, or more correctly, reduced to only creating 
a 3D model of the target (vs. the tooth and target) and 
proceeding from there. 
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Claims 

1. A method for creating a dental model from a series 
of images of an intra-oral object, said method com- 

5 prising the steps of: 

(a) capturing a series of images of an intra-oral 
object from a plurality of capture positions, 
where the object includes common surface fea- 

10 tures and a control target arranged with respect 

to the object to provide control features; 

(b) measuring the control features from the im- 
ages of the control target included with the im- 
ages of the object; 

15 (c) analytically generating a 3-dimensional 

model of the object by photogrammetrically 
aligning the measurements of the control fea- 
tures, thereby providing a photogrammetrically 
aligned 3-dimensional model of the object while 
20 reducing image errors due to the variable ori- 

entations of the capture positions; and 
(d) adjusting the photogrammetrically aligned 
3-dimensional mode! of the object by aligning 
the common features of the model to like fea- 
25 tures on an image of the object, thereby pro- 

ducing an aligned dental model from the series 
of images. 

2. The method as claimed in claim 1 wherein step (b) 
30 further includes the step of measuring the common 

features from the series of images of the object. 

3. The method as claimed in claim 1 wherein step (c) 
comprises the steps of: 

35 

performing a photog ram metric adjustment; and 
refining the photogrammetric adjustment by 
photogrammetrically projecting a 3-dimension- 
al model of the image, detemnining misalign- 
40 ment of the control features and connecting the 

misalignment, thereby producing the photo- 
grammetrically aligned 3-dimensional model of 
the object. 

45 4. The method as claimed In claim 1 wherein step (d) 
comprises the steps of: 



determining misalignment of the common fea- 
tures in the photogrammetrically aligned 3-di- 
50 mensional model relative to the images of the 

object by photogrammetrically projecting the 
model onto an image of the object; and 
applying a 3-dimensional morphing algorithm 
to correct for the misalignment. 

55 

5. The method as claimed in claim 1 further compris- 
ing the steps of providing a database of generic 
3-dimensional models and utilizing a selected one 
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of the generic models in step (d) in the alignment of 
the common features of the aligned dental model to 
like features on the image of the object. 

6. A system for creating a dental model from a series 5 
of Images of an intra-oral object, said system com- 
prising: 

a camera for capturing a series of images of an 
Intra-oral object from a plurality of capture po- io 
sitlons, where the object includes common sur- 
face features and a control target arranged with 
respect to the object to provide control features; 
photogrammetric means for measuring the 
control features from the images of the control ^5 
target included with the images of the object; 
a digital processor including instructions for (a) 
analytically generating a 3-dimenslonal model 
of the object by photogrammetrlcally aligning 
the measurements of the control features, 
thereby providing a photo gram metrically 
aligned 3-dimensional model of the object while 
reducing image errors due to the variable ori- 
entations of the capture positions; and (b) ad- 
justing the photogrammetrically aligned 3-di- 
mensional model of the object by aligning the 
common features of the model to like features 
on an Image of the object, thereby producing 
an aligned dental model from the series of im- 
ages. 30 

7. The system as claimed in claim 6 wherein said pho- 
togrammetric means further measures the common 
features from the series of Images of the object. 

35 

8. The system as claimed in claim 6 wherein said dig- 
ital processor further Includes instructions for per- 
forming a photogrammetric adjustment and refining 
the photogrammetric adjustment by photogrammet- 
rically projecting a 3-dimensional model of the im- 40 
age, determining misalignment of the control fea- 
tures and correcting the misalignment, thereby pro- 
ducing the photogrammetrically aligned 3-dimen- 
sional model of the object. 

45 

9. The system as claimed in claim 6 wherein said dig- 
ital processor further Includes instructions for deter- 
mining misalignment of the common features In the 
photogrammetrically aligned 3-dimensional model 
relative to the images of the object by photogram- 50 

' metrically projecting the model onto an image of the 
object and applying a 3-dlmensional morphing al- . 
gorithm to correct for the misalignment. 
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